1. Field of the Invention
This invention relates to an image processing method and an image processing system, and more particularly to an image processing method of and an image processing system for processing an image to enhance sharpness while suppressing granularity (high frequency noise).
2. Description of the Related Art
There has been known a system in which an image signal is obtained by reading an image on a photographic film or a printed image (including a color image) by an image sensor such as a CCD, the image signal is variously processed and an image is printed or displayed on a CRT or the like on the basis of the processed image signal. Such image processing includes, for instance, frequency processing which acts on predetermined frequency components of the image, and gradation processing which acts on the density of the image.
As the frequency processing, sharpness enhancement processing for suppressing blur of the edges of an image, granularity suppressing processing such as smoothening processing for suppressing granularity due to the granularity value of the photosensitive material and the like have been known.
As the sharpness enhancement processing, there have been known unsharp masking processing, high frequency band enhancement filter processing, processing for enhancing high frequency components of the image while suppressing intermediate frequency components (Japanese Unexamined Patent Publication No. 9(1997)-22460), and the like. As the granularity suppressing processing, there have been known, for instance, a median filter processing, hysteresis smoothing processing, noise removing processing by repetition, and granularity suppressing processing using morphology operation.
When the sharpness enhancement processing is carried out, granularity of the image is also enhanced though the sharpness of the image is improved, whereas when the granularity suppressing processing is carried out, the sharpness of the image is deteriorated though the granularity of the image is suppressed.
Accordingly, there has been a demand for image processing which can enhance the sharpness of the image while suppressing the granularity of the image, and there have been proposed various processings which can enhance the sharpness of the image and at the same time can suppress the granularity of the image as disclosed, for instance, in U.S. Pat. No. 4,812,903 and Japanese Unexamined Patent Publication Nos. 63(1988)-26783 and 9(1997)-22460. However, it should be said that the conventional techniques fail in effectively realizing enhancement of the sharpness together with suppression of the granularity.
For example, the technique disclosed in Japanese Unexamined Patent Publication No. 9(1997)-22460 tries to realize enhancement of the sharpness together with suppression of the granularity by enhancing high frequency components which affect the sharpness of the image and at the same time suppressing intermediate frequency components which affect the granularity of the image. This technique utilizes the fact that the granular portion is low in the degree of correlation with color and the image signal (edge) portion is high in the degree of correlation with color. In this technique, the image is developed into frequency ranges and the strength of intermediate to high frequency components is controlled so that the portion which is low in the degree of correlation with color is less enhanced, thereby suppressing the granularity. However in this processing, since continuity between adjacent pixels in a real image is not taken into account, abrupt change in the degree of control occurs in some part of the image, where the granularity cannot be sufficiently suppressed. Accordingly, this processing fails in effectively suppressing the granularity though it can effectively enhance the sharpness.
In view of the foregoing observations and description, the primary object of the present invention is to provide an image processing method and system which can effectively realize enhancement of the sharpness together with suppression of the granularity.
The image processing method and system of the present invention are characterized in that processing for suppressing granularity and processing for enhancing sharpness are separately carried out on an original image signal to obtain a granularity-suppressed image signal and a sharpness-enhanced image signal, at least one intermediate processed image signal is obtained by processing the image signal components of the granularity-suppressed image signal and the sharpness-enhanced image signal as well as the original image signal, as required, in one-to-one correspondence, and as the image signal component for each pixel of the final processed image signal, the image signal component for the corresponding pixel of an image signal selected from the original image signal and the intermediate processed image signal(s) is taken according to an indicator value defined by the granularity-suppressed image signal, the sharpness-enhanced image signal and the original image signal.
That is, in accordance with one aspect of the present invention, there is provided a method of processing an original image signal fin made up of a plurality of image signal components each representing a pixel of an original image (inclusive of a color image) to suppress the granularity of the original image and enhance the sharpness of the original image, and thereby obtaining a final processed image signal fout, the method comprising the steps of
carrying out processing for suppressing granularity and processing for enhancing sharpness separately on the original image signal fin to obtain a granularity-suppressed image signal f1 and a sharpness-enhanced image signal f2,
obtaining at least one intermediate processed image signal f3 (f4, f5, . . . ) by operation based on at least the granularity-suppressed image signal and the sharpness-enhanced image signal, and
taking, as the image signal component for each pixel of the final processed image signal fout, the image signal component for the corresponding pixel of an image signal selected from the intermediate processed image signal(s) f3 and the original image signal fin according to an indicator value k(f1, f2, fin) calculated on the basis of the granularity-suppressed image signal f1, the sharpness-enhanced image signal f2 and the original image signal fin.
In this specification, fin, f1, f2, fout and the like are sometimes used to denote image signal components of the corresponding image signals, which will be apparent to those skilled in the art. As the indicator value k(f1, f2, fin), for instance, (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) may be used. In the case where one intermediate processed image signal f3 is obtained, the image signal component of the original image signal fin is taken as the image signal component of the final processed image signal fout when the indicator value (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) is smaller than 0, and the image signal component of the intermediate processed image signal f3 is taken as the image signal component of the final processed image signal fout when the indicator value (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) is not smaller than 0 as represented by the following formula (1).
In the case where a pair of intermediate processed image signals f3 and f4 are obtained, the image signal component of the first intermediate processed image signal f3 is taken as the image signal component of the final processed image signal fout when the indicator value (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) is smaller than 0, and the image signal component of the second intermediate processed image signal f4 is taken as the image signal component of the final processed image signal fout when the indicator value (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) is not smaller than 0 as represented by the following formula (4).                               f          out                =                  {                                                                                                                f                      in                                        ⁢                                          xe2x80x83                                        ⁢                                          (                                                                        when                          ⁢                                                      xe2x80x83                                                    ⁢                                                                                    (                                                              f1                                -                                                                  f                                  in                                                                                            )                                                        ·                                                          (                                                              f2                                -                                                                  f                                  in                                                                                            )                                                                                                       less than                         0                                            )                                                        ,                                                                                                      f3                  ⁢                                      xe2x80x83                                    ⁢                                      (                                                                  when                        ⁢                                                  xe2x80x83                                                ⁢                                                                              (                                                          f1                              -                                                              f                                in                                                                                      )                                                    ·                                                      (                                                          f2                              -                                                              f                                in                                                                                      )                                                                                              ≧                      0                                        )                                                                                                          (        1        )            
wherein fin represents the original image signal, f1 represents the granularity-suppressed image signal, f2 represents the sharpness-enhanced image signal, and f3 represents the intermediate processed image signal.                               f          out                =                  {                                                                                          f3                    ⁢                                          xe2x80x83                                        ⁢                                          (                                                                        when                          ⁢                                                      xe2x80x83                                                    ⁢                                                                                    (                                                              f1                                -                                                                  f                                  in                                                                                            )                                                        ·                                                          (                                                              f2                                -                                                                  f                                  in                                                                                            )                                                                                                       less than                         0                                            )                                                        ,                                                                                                      f4                  ⁢                                      xe2x80x83                                    ⁢                                      (                                                                  when                        ⁢                                                  xe2x80x83                                                ⁢                                                                              (                                                          f1                              -                                                              f                                in                                                                                      )                                                    ·                                                      (                                                          f2                              -                                                              f                                in                                                                                      )                                                                                              ≧                      0                                        )                                                                                                          (        4        )            
wherein fin represents the original image signal, f1 represents the granularity-suppressed image signal, f2 represents the sharpness-enhanced image signal, f3 represents the first intermediate processed image signal, and f4 represents the second intermediate processed image signal.
That is, formula (1) represents processing in which the image signal component of the intermediate processed image signal f3 is taken as the image signal component of the final processed image signal fout when both the image signal components of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 are larger or smaller than the image signal component of the original image signal fin or when the image signal component of at least one of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 is equal to the image signal component of the original image signal fin, and the image signal component of the original image signal fin is taken as the image signal component of the final processed image signal fout when the image signal component of one of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 is smaller than the image signal component of the original image signal fin and the image signal component of the other of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 is larger than the image signal component of the original image signal fin. At portions such as an edge portion of an image where the density abruptly changes, the indicator value (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) is generally not smaller than 0 ((f1xe2x88x92fin)xc2x7(f2xe2x88x92fin)xe2x89xa70), and accordingly the intermediate processed image signal f3 where the sharpness is higher than the original image signal fin is selected. On the other hand, at the other portions (e.g., flat density portions), the indicator value (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) is generally smaller than 0 ((f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) less than 0), and accordingly, the original image signal fin is selected since sufficient granularity suppressing effect cannot be obtained with the intermediate processed image signal f3 which is a weighted average of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2. In this case, by selecting the original image signal fin, a granularity-suppressed final processed image signal fout can be obtained. Thus, the granularity of the image can be suppressed without deteriorating the sharpness of the edge portion and the like.
In the case where three intermediate processed image signals f3, f4 and f5 are obtained, the image signal component of one of the first to third intermediate processed image signal f3, f4 and f5 is taken as the image signal component of the final processed image signal fout according to whether the indicator value k(f1, f2, fin) is positive or negative and the value of |finxe2x88x92f1| as represented by the following formula (15).
fout=f3 (when |finxe2x88x92f1|xe2x89xa7Th)
fout=f4 (when |finxe2x88x92f1| less than Th and (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin) less than 0)
fout=f5 (when |finxe2x88x92f1| less than Th and (f1xe2x88x92fin)xc2x7(f2xe2x88x92fin)xe2x89xa70)xe2x80x83xe2x80x83(15)
wherein fin represents the original image signal, f1 represents the granularity-suppressed image signal, f2 represents the sharpness-enhanced image signal, f3 represents the first intermediate processed image signal, f4 represents the second intermediate processed image signal, f5 represents the third intermediate processed image signal, and Th represents a constant.
The operation for obtaining the intermediate processed image signal f3 may be selected from a plurality of kinds of preset operations according to the relation between the intermediate processed image signal f3 and the original image signal fin, the kind of the image and/or the position of the pixel, or may be input from the external for each processing. Specifically the operation is preferably that defined by the following formula (2) which is based on the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2, or that defined by the following formula (3) which is based on the granularity-suppressed image signal f1, the sharpness-enhanced image signal f2 and the original image signal fin.
f3=w1xc2x7f1+w2xc2x7f2xe2x80x83xe2x80x83(2)
wherein w1+w2=1 (0 less than w1 less than 1, 0 less than w2 less than 1), f1 represents the granularity-suppressed image signal, f2 represents the sharpness-enhanced image signal and f3 represents the intermediate processed image signal.
f3={|f1xe2x88x92fin|(f1xe2x88x92fin)+|f2xe2x88x92fin|(f2xe2x88x92fin)}/{|f1xe2x88x92fin|+|f2xe2x88x92fin|}+finxe2x80x83xe2x80x83(3)
wherein fin represents the original image signal, f1 represents the granularity-suppressed image signal, f2 represents the sharpness-enhanced image signal and f3 represents the intermediate processed image signal.
That is, in accordance with formula (2), the intermediate processed image signal f3 is obtained by weighted addition of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2, and by weighted average of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2, the sharpness can be enhanced while suppressing the granularity. Also in this operation, the weighting factors w1 and w2 may be changed pixel to pixel according to the relation between the intermediate processed image signal f3 and the original image signal fin, or the like, or may be common to all the pixels.
Further each of the weighting factors w1 and w2 is preferably 0.2 to 0.8, more preferably 0.3 to 0.7 and most preferably 0.4 to 0.6.
In accordance with formula (3), the image signal components of the sharpness-enhanced image signal f2 are taken as the image signal components of the intermediate processed image signal f3 for pixels where the image signal components of the granularity-suppressed image signal f1 are equal to those of the original image signal fin, the image signal components of the granularity-suppressed image signal f1 are taken as the image signal components of the intermediate processed image signal f3 for pixels where the image signal components of the sharpness-enhanced image signal f2 are equal to those of the original image signal fin, and values obtained by weighting the differences between the image signal components of the granularity-suppressed image signal f1 and the original image signal fin and between the image signal components of the sharpness-enhanced image signal f2 and the original image signal f1 and adding the weighted differences are taken as the image signal components of the intermediate processed image signal f3 for pixels where the image signal components of neither of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 are equal to those of the original image signal fin. With this operation, for the image signal components for pixels in the flat density portion (an area of the image where change of the density or brightness is less) where the granularity is more apt to be remarked than the sharpness, the image signal components of the granularity-suppressed image signal f1 is more weighted and accordingly the granularity can be suppressed. To the contrast, for the image signal components for pixels in the edge portion (an area of the image where change of the density or brightness is sharp) where the sharpness is more apt to be remarked than the granularity, the image signal components of the sharpness-enhanced image signal f2 is more weighted and accordingly the sharpness can be enhanced. This operation is a specific example of the operation where processing is selected pixel to pixel according to the relation with the original image signal fin.
Formula (4) represents processing in which the image signal component of the second intermediate processed image signal f4 is taken as the image signal component of the final processed image signal fout when both the image signal components of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 are larger or smaller than the image signal component of the original image signal fin or when the image signal component of at least one of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 is equal to the image signal component of the original image signal fin (kxe2x89xa70), and the image signal component of the first intermediate processed image signal f3 is taken as the image signal component of the final processed image signal fout when the image signal component of one of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 is smaller than the image signal component of the original image signal fin and the other of the image signal components of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 is larger than the image signal component of the original image signal fin (k less than 0).
It is preferred that the degree of granularity suppression of the first intermediate processed image signal f3 be higher than that of the second intermediate processed image signal f4, and the degree of sharpness enhancement of the second intermediate processed image signal f4 be higher than that of the first intermediate processed image signal f3. At portions such as an edge portion of an image where the density abruptly changes, the indicator value k is generally not smaller than 0 (kxe2x89xa70), and accordingly by selecting the second intermediate processed image signal f4 which is higher than the first intermediate processed image signal f3 in the degree of sharpness enhancement, the edge portion can be prevented from becoming unsharp. On the other hand, at the other portions (e.g., flat density portions), the indicator value k is generally smaller than 0 (k less than 0) and the granularity is apt to be remarked, and accordingly, by selecting the first intermediate processed image signal f3 which is higher than the second intermediate processed image signal f4 in the degree of granularity suppression, the granularity can be suppressed.
It is preferred that the first and second intermediate processed image signals f3 and f4 be obtained by weighting the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 with weighting factors w1a, w1b, w2a and w2b and adding the weighted image signals f1 and f2 as represented by the following formulae (5) and (6).
f3=w1axc2x7f1+w1bxc2x7f2xe2x80x83xe2x80x83(5)
f4=w2axc2x7f1+w2bxc2x7f2xe2x80x83xe2x80x83(6)
wherein w1a+w1b=1 (0xe2x89xa6w1axe2x89xa61, 0xe2x89xa6w1bxe2x89xa61), w2a+w2b=1 (0xe2x89xa6w2axe2x89xa61, 0xe2x89xa6w2bxe2x89xa61), f1 represents the granularity-suppressed image signal, f2 represents the sharpness-enhanced image signal, f3 represents the first intermediate processed image signal, and f4 represents the second intermediate processed image signal.
It is preferred that the weighting factors w1a, w1b, w2a and w2b in the formulae (5) and (6) be changed depending on the original image signal fin. For example, in the case of an original image signal fin which is set so that the density is increased as the value of the image signal component increases, it is preferred that the weighting factors w1a, w1b, w2a and w2b be defined by the following formulae (7) to (10), or by the following formulae (11) to (14).
w1a=1xe2x88x92(cxcex1+t)/Cxe2x80x83xe2x80x83(7)
w1b=(cxcex1+t)/Cxe2x80x83xe2x80x83(8)
w2a=(cxcex1+t)/Cxe2x80x83xe2x80x83(9)
w2b=1xe2x88x92(cxcex1+t)/Cxe2x80x83xe2x80x83(10)
wherein
t=(|f2xe2x88x92fin|)/{(|f1xe2x88x92fin|)(|f2xe2x88x92fin|)},
xcex1=fin/f0,
fin=an original image signal,
f0=a reference value,
c=a constant, and
C=a constant.
w1a={c(1xe2x88x92xcex1)+t)}/Cxe2x80x83xe2x80x83(11)
w1b=1xe2x88x92{c(1xe2x88x92xcex1)+t)}/Cxe2x80x83xe2x80x83(12)
w2a=1xe2x88x92(cxcex1+t)/Cxe2x80x83xe2x80x83(13)
w2b=(cxcex1+t)/Cxe2x80x83xe2x80x83(14)
wherein
t=(|f2xe2x88x92fin|)/{(|f1xe2x88x92fin|)(|f2xe2x88x92fin|)},
xcex1=fin/f0,
fin=an original image signal,
f0=a reference value,
c=a constant, and
C=a constant.
That is, according to the formulae (7) to (10), or the formulae (11) to (14), the weighting factors w1a and w1b for the first intermediate processed image signal f3 are set so that the weight on the granularity-suppressed image signal f1 becomes larger as the image signal component of the original image signal fin becomes smaller, and the weighting factors w2a and w2b for the second intermediate processed image signal f4 are set so that the weight on the sharpness-enhanced image signal f2 becomes larger as the image signal component of the original image signal fin becomes smaller.
When the weighting factors are set in this manner, in the first intermediate processed image signal f3 which is selected as the final processed image signal fout for a flat density portion according to formula (4), the granularity is more suppressed at a high density portion than a low density portion. Since the granularity is generally bad at a high density portion in an image, the granularity can be effectively suppressed by more suppressing the granularity at a high density portion.
The weighting factors w1a, w1b, w2a and w2b set according to formulae (11) to (14) are preferably applied to a region where the granularity is very bad. In this case, it is preferred that the constants c and C be set in a range cxe2x89xa6C less than (c+1) or c less than Cxe2x89xa6(c+1) in a region where the values of the image signal components of the original image signal fin are relatively small so that w1a greater than w1b and w2a greater than w2b are satisfied, i.e., so that the weight on the granularity-suppressed image signal f1 is larger than that on the sharpness-enhanced image signal f2 in the region for both the intermediate processed image signals f3 and f4 (provided that the first intermediate processed image signal f3 is higher than the second intermediate processed image signal f4 in the degree of granularity suppression).
The reference value f0, which determines xcex1 in the formulae (7) to (14), may be, for instance, a maximum value of the image signal components of the original image signal fin (1023 in the case where the original image signal fin is expressed in 10 bits) or 90% of the maximum value. Further it is preferred that the constant C is larger than the constant c.
Formula (15) represents processing in which the image signal component of the first intermediate processed image signal f3 is taken as the image signal component of the final processed image signal fout when the difference (|finxe2x88x92f1|) between the image signal components of the original image signal fin and the granularity-suppressed image signal f1 is not smaller than a predetermined threshold value Th, the image signal component of the third intermediate processed image signal f5 is taken as the image signal component of the final processed image signal fout when the difference (|finxe2x88x92f1|) is smaller than the threshold value Th and the image signal components of both the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 are not smaller than the image signal component of the original image signal fin (kxe2x89xa70), and the image signal component of the second intermediate processed image signal f4 is taken as the image signal component of the final processed image signal fout when the difference (|finxe2x88x92f1|) is smaller than the threshold value Th, the image signal component of one of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 is larger than that of the original image signal fin, and the image signal component of the other of the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2 is smaller than that of the original image signal fin (k less than 0).
In this case, the first intermediate processed image signal f3 is generated as an image signal having a sharpness between that of the original image signal fin and that of the sharpness-enhanced image signal f2.
According to formula (15), the first intermediate processed image signal f3 which is substantially equivalent in sharpness to the original image signal fin in which the granularity is not suppressed or to the sharpness-enhanced image signal f2 in which the sharpness is relatively high, is selected for image portions such as an image of catchlights (external light reflected at a pupil) where the difference (|finxe2x88x92f1|) is larger than the threshold value Th is selected, whereby the image portions such as the image of catchlights can be left there and the image can be rich in expression.
It is preferred that the second intermediate processed image signal f4 be higher in the degree of granularity suppression than the third intermediate processed image signal f5 and the third intermediate processed image signal f5 be higher in the degree of sharpness enhancement than the second intermediate processed image signal f4. At portions such as an edge portion of an image where the density abruptly changes, the indicator value k is generally not smaller than 0 (k greater than 0), and accordingly by selecting the third intermediate processed image signal f5 which is higher than the second intermediate processed image signal f4 in the degree of sharpness enhancement, the edge portion can be prevented from becoming unsharp. On the other hand, at the other portions (e.g., flat density portions), the indicator value k is generally smaller than 0 (k less than 0) and the granularity is apt to be remarked, and accordingly, by selecting the second intermediate processed image signal f4 which is higher than the third intermediate processed image signal f5 in the degree of granularity suppression, the granularity can be suppressed.
It is preferred that the first to third intermediate processed image signals f3, f4 and f5 be obtained by weighting the granularity-suppressed image signal f1 (or the original image signal fin) and the sharpness-enhanced image signal f2 with weighting factors w3a, w3b, w4a, w4b, w5a and w5b, and adding the weighted image signal f1 (or the weighted original image signal fin) and the weighted image signal f2 as represented by the following formulae (16) to (18).
f3=w3axc2x7f1+w3bxc2x7f2xe2x80x83xe2x80x83(16)
f4=w4axc2x7f1+w4bxc2x7f2xe2x80x83xe2x80x83(17)
f5=w5axc2x7f1+w5bxc2x7f2xe2x80x83xe2x80x83(18)
wherein w3a+w3b=1 (0xe2x89xa6w3axe2x89xa61, 0xe2x89xa6w3bxe2x89xa61), w4a+w4b=1 (0xe2x89xa6w4axe2x89xa61, 0xe2x89xa6w4bxe2x89xa61), w5a+w5b=1 (0xe2x89xa6w5axe2x89xa61, 0xe2x89xa6w5bxe2x89xa61), f1 represents the granularity-suppressed image signal, f2 represents the sharpness-enhanced image signal, f3 represents the first intermediate processed image signal, f4 represents the second intermediate processed image signal, and f5 represents the third intermediate processed image signal.
It is preferred that the weighting factors w4a, w4b, w5a and w5b in the formulae (16) to (18) be changed depending on the original image signal fin. For example, in the case of an original image signal fin which is set so that the density is increased as the value of the image signal component increases, it is preferred that the weighting factors w4a, w4b, w5a and w5b be defined by the following formulae (19) to (22), or by the following formulae (23) to (26).
w4a=1xe2x88x92(cxcex1+t)/Cxe2x80x83xe2x80x83(19)
w4b=(cxcex1+t)/Cxe2x80x83xe2x80x83(20)
w5a=(cxcex1+t)/Cxe2x80x83xe2x80x83(21)
w5b=1xe2x88x92(cxcex1+t)/Cxe2x80x83xe2x80x83(22)
wherein
t=(|f2xe2x88x92fin|)/{(|f1xe2x88x92fin|)(|f2xe2x88x92fin|)},
xcex1=fin/f0,
fin=an original image signal,
f0=a reference value,
c=a constant, and
C=a constant.
w4a={c(1xe2x88x92xcex1)+t)}/Cxe2x80x83xe2x80x83(23)
w4b=1xe2x88x92{c(1xe2x88x92xcex1)+t)}/Cxe2x80x83xe2x80x83(24)
w5a=1xe2x88x92(cxcex1+t)/Cxe2x80x83xe2x80x83(25)
w5b=(cxcex1+t)/Cxe2x80x83xe2x80x83(26)
wherein
t=(|f2xe2x88x92fin|)/{(|f1xe2x88x92fin|)(|f2xe2x88x92fin|)},
xcex1=fin/f0,
fin=an original image signal,
f0=a reference value,
c=a constant, and
C=a constant.
That is, according to the formulae (19) to (22), or the formulae (23) to (26), the weighting factors w4a and w4b for the second intermediate processed image signal f4 are set so that the weight on the granularity-suppressed image signal f1 becomes larger as the image signal component of the original image signal fin becomes smaller, and the weighting factors w5a and w5b for the third intermediate processed image signal f5 are set so that the weight on the sharpness-enhanced image signal f2 becomes larger as the image signal component of the original image signal fin becomes smaller.
When the weighting factors are set in this manner, in the second intermediate processed image signal f4 which is selected as the final processed image signal fout for a flat density portion according to formula (15), the granularity is more suppressed at a high density portion than a low density portion. Since the granularity is generally bad at a high density portion in an image, the granularity can be effectively suppressed by more suppressing the granularity at a high density portion.
The weighting factors w4a, w4b, w5a and w5b set according to formulae (23) to (26) are preferably applied to a region (e.g., extremely underexposed portion) where the granularity is very bad. In this case, it is preferred that the constants c and C be set in a range cxe2x89xa6C less than (c+1) or c less than Cxe2x89xa6(c+1) in a region where the values of the image signal components of the original image signal fin are relatively small so that w4a greater than w4b and w5a greater than w5b are satisfied, i.e., so that the weight on the granularity-suppressed image signal f1 is larger than that on the sharpness-enhanced image signal f2 in the region for both the intermediate processed image signals f4 and f5 (provided that the second intermediate processed image signal f4 is higher than the third intermediate processed image signal f5 in the degree of granularity suppression).
The reference value f0, which determines a in the formulae (19) to (26), may be, for instance, a maximum value of the image signal components of the original image signal fin (1023 in the case where the original image signal fin is expressed in 10 bits) or 90% of the maximum value. Further it is preferred that the constant C is larger than the constant c.
As the granularity suppressing image processing, there can be used, for instance, a median filter processing, hysteresis smoothing processing, noise removing processing by repetition, and granularity suppressing processing (smoothing) using morphology operation. As the sharpness enhancement image processing, there can be used, for instance, unsharp masking processing, high frequency band enhancement filter processing, and processing in which an image signal is separated into a low frequency component, an intermediate frequency component and a high frequency component, the high frequency component is enhanced while the intermediate frequency component is suppressed, and the processed high frequency component, the processed intermediate frequency component and the low frequency component are combined into an image signal (Japanese Unexamined Patent Publication No. 9(1997)-22460). However a combination of the granularity suppressing processing using morphology operation and the processing in which an image signal is separated into a low frequency component, an intermediate frequency component and a high frequency component, the high frequency component is enhanced while the intermediate frequency component is suppressed, and the processed high frequency component, the processed intermediate frequency component and the low frequency component are combined into an image signal (Japanese Unexamined Patent Publication No. 9(1997)-22460) is optimal.
Though the morphology operation is generally developed as a set theory in an N-dimensional space, it is often applied to images which are two-dimensional spaces. See, for instance, Japanese Unexamined Patent Publication Nos. 8(1996)-272961, 9(1997)-248291 and 9(1997)-91421. The morphology operation will be briefly described on a light and shade image by way of example.
Considering a light and shade image as a space where a point (x, y) has a height corresponding to a density value f(x, y), and a one-dimensional function f(x) corresponding a cross-section of the space will be discussed.
It is assumed that structure element g employed in the morphology operation is a symmetric function symmetrical about the origin as represented by the following formula (28) and is 0 in value in its domain G which is defined by the following formula (29).
g5(X)=g(xe2x88x92X)xe2x80x83xe2x80x83(28)
G={xe2x88x92m, xe2x88x92m+1, . . . , xe2x88x921, 0, 1, . . . , mxe2x88x921, m}xe2x80x83xe2x80x83(29)
At this time, the basic of the morphology operation is very simple as represented by the following formula (30) to (33).
dilation;[f⊕G5](i)=max{f(ixe2x88x92m), . . . , f(i), . . . , f(i+m)}xe2x80x83xe2x80x83(30)
erosion;[fxe2x8ax96G5](i)=min{f(ixe2x88x92m), . . . , f(i), . . . , f(i+m)}xe2x80x83xe2x80x83(31)
opening;fg=(fxe2x8ax96g5)⊕gxe2x80x83xe2x80x83(32)
closing;fg=(f⊕g5)xe2x8ax96gxe2x80x83xe2x80x83(33)
The dilation processing is for searching the maximum value in a range of xc2x1m (a value determined depending on structure element B and corresponding the mask size in FIGS. 15A to 15D) about an object pixel (See, FIG. 15A), while the erosion processing is for searching the minimum value in the range of xc2x1m about the object pixel (See, FIG. 15B). The opening processing is for performing the dilation processing after the erosion processing, that is, for searching the value maximum after search of the minimum value, and the closing processing is for performing the erosion processing after the dilation processing, that is, for searching the minimum value after search of the maximum value.
In other words, the opening processing smoothes the density curve f(X) from the low density side and suppresses density projection portions (portions higher in density than a surrounding portion) at which the density fluctuates within a range which is spatially narrower than the mask size 2m (FIG. 15C). To the contrast, the closing processing smoothes the density curve f(X) from the high density side and suppresses density recession portions (portions lower in density than a surrounding portion) at which the density fluctuates within a range which is spatially narrower than the mask size 2m (FIG. 15D).
In a high-density high-signal-level image signal, the signal level increases as the density increases whereas in a high-brightness high-signal-level image signal, the signal level increases as the brightness increases. Accordingly, the dilation processing for the high-density high-signal-level image signal is equivalent to the erosion processing for the high-brightness high-signal-level image signal (FIG. 15B), the erosion processing for the high-density high-signal-level image signal is equivalent to the dilation processing for the high-brightness high-signal-level image signal (FIG. 15A), the opening processing for the high-density high-signal-level image signal is equivalent to the closing processing for the high-brightness high-signal-level image signal (FIG. 15D), and the closing processing for the high-density high-signal-level image signal is equivalent to the opening processing for the high-brightness high-signal-level image signal (FIG. 15C).
By carrying out the opening processing or the closing processing by the morphology operation on an image signal representing an original image, granularity of the image (noise in the image signal) can be suppressed or removed. (xe2x80x9cMorphologyxe2x80x9d by Obata, CORONA)
When granularity suppressing processing using morphology operation is employed in the image processing method of the present invention, it is preferred that granularity suppressing defined by the following formula (27) be employed.                     f1        =                  {                                                                                          f                    opn                                    ⁢                                      xe2x80x83                                    ⁢                                      (                                                                  when                        ⁢                                                  xe2x80x83                                                ⁢                                                  f                          cls                                                                    =                                              f                        in                                                              )                                                                                                                                            f                    cls                                    ⁢                                      xe2x80x83                                    ⁢                                      (                                                                  when                        ⁢                                                  xe2x80x83                                                ⁢                                                  f                          opn                                                                    =                                              f                        in                                                              )                                                                                                                                                                  (                                                                        f                          opn                                                +                                                  f                          cls                                                                    )                                        /                    2                                    ⁢                                      xe2x80x83                                    ⁢                                      (                                                                                            when                          ⁢                                                      xe2x80x83                                                    ⁢                                                      f                            cls                                                                          ≠                                                  f                          in                                                                    ,                                              xe2x80x83                                            ⁢                                                                        and                          ⁢                                                      xe2x80x83                                                    ⁢                                                      f                            opn                                                                          ≠                                                  f                          in                                                                                      )                                                                                                          (        27        )            
wherein
f1: granularity-suppressed image signal,
fcls: closing image signal,
fopn: opening image signal,
fin: original image signal.
When granularity suppressing processing using morphology operation is employed, portions such as an edge portion of an image where the density abruptly changes are substantially perfectly reserved (kept as they are), and accordingly, (f1xe2x88x92fin)) in formulae (1), (4) and (15) is substantially constantly not smaller than 0 and the indicator value k is not smaller than 0.
The sharpness enhancement processing by enhancing high frequency components of the image while suppressing intermediate frequency components disclosed in Japanese Unexamined Patent Publication No. 9(1997)-22460will be briefly described, hereinbelow.
In this processing, an image signal is separated into a low frequency component, an intermediate frequency component and a high frequency component, enhancement/suppression processing in which the high frequency component is enhanced and the intermediate frequency component is suppressed is carried out, and the processed high frequency component, the processed intermediate frequency component and the low frequency component are combined into a final processed image signal fout.
The low frequency component, intermediate frequency component and high frequency component are those distributed as shown in FIG. 16. The intermediate frequency component is distributed with a peak near ⅓ of a Nyquist frequency of output for reproducing the final processed image signal fout as a visible image, the low frequency component is distributed with a peak at a frequency at which the Nyquist frequency of the output becomes 0, and the high frequency component is distributed with a peak at the Nyquist frequency of the output. The sum of the low frequency component, the intermediate frequency component and the high frequency component is 1 at each frequency.
In this processing, it is preferred that brightness components be extracted from the separated high and intermediate frequency components and said enhancement/suppression processing and said combination processing be carried out on the basis of only the brightness components.
Further, in this processing, it is preferred that a particular color region of the image be extracted and the intermediate component fraction corresponding to the particular color region be further suppressed in the enhancement/suppression processing.
Further, in this processing, it is preferred that an evaluation value of the intermediate frequency component and/or the high frequency component be obtained after the image signal is separated into the low frequency component, the intermediate frequency component and the high frequency component, and the intermediate frequency component for a pixel the evaluation value for which is smaller than a predetermined threshold value be more suppressed in the enhancement/suppression processing than the intermediate frequency component for a pixel the evaluation value for which is larger than the predetermined threshold value. In this case, it is further preferred that the high frequency component for the pixel the evaluation value for which is smaller than the predetermined threshold value be less enhanced in the enhancement/suppression processing than the high frequency component for the pixel the evaluation value for which is larger than the predetermined threshold value.
The xe2x80x9cevaluation valuexe2x80x9d as used here means, for instance, at least one of correlation values between twos of red, green and blue, a value of local dispersion of the frequency components of the image signal.
Further, it is preferred that the evaluation value be at least one of correlation values between twos of red, green and blue of the intermediate frequency component and/or the high frequency component for the corresponding pixels.
The evaluation value may be processed with a median filter and the enhancement/suppression processing may be performed on the basis of comparison between the processed evaluation value and said threshold value. Further, the evaluation value of the intermediate frequency component and/or the high frequency component may be calculated on the basis of different colors of red, green and blue.
In granularity suppression in the enhancement/suppression processing of Japanese Unexamined Patent Publication No. 9(1997)-22460, continuity between adjacent pixels in a real image is not taken into account as described above. Accordingly, it is difficult to recognize the granularity suppression effect of the enhancement/suppression processing. To the contrast, the granularity suppression processing based on the morphology operation is processing on a real image space using isolation of granularity (continuity of noise between adjacent pixels is low). Accordingly, when an image signal is processed by this method, continuity between adjacent pixels is improved and the degree of suppression of granularity comes to be more easy to visually recognize.
It is preferred that the original image signal be so-called raw data, data as read out from a recording medium such as a print or a photographic film by a scanner or the like. However the original image signal may be an image signal which has been subjected to gradation processing, color correction, shading correction and/or the like. Further the original image signal need not be an image signal read out by a scanner or the like but may be an image signal such as directly taken out from a digital camera.
In accordance with another aspect of the present invention, there is provided an image processing system for carrying out the image processing method of the present invention. That is, the image processing system of the present invention is for processing an original image signal made up of a plurality of image signal components each representing a pixel of an original image and obtaining a final processed image signal, and comprises,
a granularity suppressing means which carries out processing for suppressing granularity on the original image signal and obtains a granularity-suppressed image signal,
a sharpness enhancing means which carries out processing for enhancing sharpness on the original image signal and obtains sharpness-enhanced image signal,
an intermediate processed image signal generating means which generates at least one intermediate processed image signal by operation based on at least the granularity-suppressed image signal and the sharpness-enhanced image signal, and
a final processed image signal generating means which obtains a final processed image signal by taking, as the image signal component for each pixel of the final processed image signal, the image signal component for the corresponding pixel of an image signal selected from the original image signal and the intermediate processed image signal(s) according to an indicator value calculated on the basis of the granularity-suppressed image signal, the sharpness-enhanced image signal and the original image signal.
As the granularity suppressing image processing to be carried out by the granularity suppressing means, there can be used, for instance, a median filter processing, hysteresis smoothing processing, noise removing processing by repetition, and granularity suppressing processing using morphology operation. As the sharpness enhancement image processing to be carried out by the sharpness enhancing means, there can be used, for instance, unsharp masking processing, high frequency band enhancement filter processing, and processing in which an image signal is separated into a low frequency component, an intermediate frequency component and a high frequency component, the high frequency component is enhanced while the intermediate frequency component is suppressed, and the processed high frequency component, the processed intermediate frequency component and the low frequency component are combined into an image signal (Japanese Unexamined Patent Publication No. 9(1997)-22460). However a combination of the granularity suppressing processing using morphology operation and the processing in which an image signal is separated into a low frequency component, an intermediate frequency component and a high frequency component, the high frequency component is enhanced while the intermediate frequency component is suppressed, and the processed high frequency component, the processed intermediate frequency component and the low frequency component are combined into an image signal (Japanese Unexamined Patent Publication No. 9(1997)-22460) is optimal.
When the granularity suppressing processing using morphology operation is employed as the granularity suppressing image processing to be carried out by the granularity suppressing means, it is preferred that the granularity suppressing processing be that defined by the aforesaid formula (27).
The operation for generating at least one intermediate processed image signal by the intermediate processed image signal generating means may be selected from a plurality of kinds of preset operations according to the relation with the original image signal, the kind of the image and/or the position of the pixel, or may be input from the external for each processing.
Specifically, when one intermediate processed image signal f3 is to be generated, the operation is preferably that defined by the aforesaid formula (2) which is based on the granularity-suppressed image signal f1 and the sharpness-enhanced image signal f2, or that defined by the following formula (3) which is based on the granularity-suppressed image signal f1, the sharpness-enhanced image signal f2 and the original image signal fin.
It is preferred that the final processed image generating means obtains a final processed image signal by taking the image signal component of one of the intermediate processed image signal f3 and the original image signal fin as the image signal component for each pixel of the final processed image signal fout according to the aforesaid formula (1).
It is preferred that when two or more intermediate processed image signals f3, f4, . . . are to be obtained, the operation defined by the aforesaid formulae (5) and (6), or (16) to (18) be applied.
It is preferred that the weighting factors w1a, w1b, w2a and w2b in the formulae (5) and (6) and the weighting factors w4a, w4b, w5a and w5b in the formulae (16) to (18) be changed depending on the original image signal fin. For example, in the case of an original image signal fin which is set so that the density is increased as the value of the image signal component increases, it is preferred that the weighting factors w1a, w1b, w2a and w2b and the weighting factors w4a, w4b, w5a and w5b be defined by the aforesaid formulae (7) to (10), (11) to (14), or (23) to (26).
Further it is preferred that the final processed image signal generating means obtains a final processed image signal fout by taking the image signal component of one of the two intermediate processed image signals f3 and f4 as the image signal component for each pixel of the final processed image signal fout according to the indicator value by the processing represented by the aforesaid formula (4), or obtains a final processed image signal fout by taking the image signal component of one of the three intermediate processed image signals f3, f4 and f5 as the image signal component for each pixel of the final processed image signal fout according to the indicator value by the processing represented by the aforesaid formula (15).
It is possible to record the steps of the image processing method of the present invention on a computer-readable recording medium as program for causing a computer to perform the steps.
That is, it is possible to provide a computer-readable recording medium on which the steps of carrying out processing for suppressing granularity and processing for enhancing sharpness separately on the original image signal to obtain a granularity-suppressed image signal and a sharpness-enhanced image signal, obtaining at least one intermediate processed image signal by operation based on at least the granularity-suppressed image signal and the sharpness-enhanced image signal, and taking the image signal component of one of the intermediate processed image signal(s) and the original image signal as the image signal component for each pixel of the final processed image signal according to the indicator value calculated on the basis of the granularity-suppressed image signal, the sharpness-enhanced image signal and the original image signal are recorded as program for causing a computer to perform the steps.
The intermediate processed image signal to be obtained may be one. In this case, the single intermediate processed image signal may be obtained by the operation processing defined by any one of the aforesaid formulae (1), (2) and (3) though the operation processing defined by the aforesaid formula (2) is preferred.
Two different intermediate processed image signals may be obtained. In this case, the two intermediate processed image signal may be obtained by the operation processing defined by the aforesaid formula (4), or by the operation processing defined by the aforesaid formulae (5) and (6). In the case where the operation processing defined by the aforesaid formulae (5) and (6) is employed, it is preferred that the weighting factors w1a, w1b, w2a and w2b in the formulae (5) and (6) be defined by the aforesaid formulae (7) to (10) or (11) to (14).
Three different intermediate processed image signals may be obtained. In this case, the three intermediate processed image signal may be obtained by the operation processing defined by the aforesaid formula (15), or by the operation processing defined by the aforesaid formulae (16) to (18). In the case where the operation processing defined by the aforesaid formulae (16) to (18) is employed, it is preferred that the weighting factors w4a, w4b, w5a and w5b in the formulae (17) and (18) be defined by the aforesaid formulae (19) to (22) or (23) to (26).
It is preferred that the steps of carrying out processing for suppressing granularity and processing for enhancing sharpness involve the granularity suppressing processing using morphology operation and the processing in which an image signal is separated into a low frequency component, an intermediate frequency component and a high frequency component, the high frequency component is enhanced while the intermediate frequency component is suppressed, and the processed high frequency component, the processed intermediate frequency component and the low frequency component are combined into an image signal.
It is further preferred that the granularity suppressing processing using morphology operation is processing defined by the aforesaid formula (27).
As can be understood from the description above, in accordance with the present invention, since the image signal component of one of the intermediate processed image signal(s) and the original image signal is taken as the image signal component for each pixel of the final processed image signal according to an indicator value calculated on the basis of the granularity-suppressed image signal, the sharpness-enhanced image signal and the original image signal, both the granularity suppression and the sharpness enhancement can be more effectively realized as compared with various conventional methods or the systems.
That is, since the intermediate processed image signal is obtained by processing the image signal components of the granularity-suppressed image signal and the sharpness-enhanced image signal as well as the original image signal, as required, in one-to-one correspondence, the intermediate processed image signal is effectively suppressed with the granularity without deteriorating the sharpness at edge portions or the like of the image. Further since the image signal component of one of the intermediate processed image signal(s) and the original image signal is taken as the image signal component for each pixel of the final processed image signal according to the indicator value which is calculated on the basis of the granularity-suppressed image signal, the sharpness-enhanced image signal and the original image signal and is indicative of the features of kinds of portions of the image, the granularity of the image further precisely suppressed without deteriorating the sharpness at edge portions or the like of the image.
Further when a combination of the granularity suppressing processing using morphology operation and the processing disclosed in Japanese Unexamined Patent Publication No. 9(1997)-22460 in which an image signal is separated into a low frequency component, an intermediate frequency component and a high frequency component, the high frequency component is enhanced while the intermediate frequency component is suppressed, and the processed high frequency component, the processed intermediate frequency component and the low frequency component are combined into an image signal, the following result can be further expected.
That is, when the operation processing represented by the aforesaid formula (3) is employed on an image signal where the granularity suppression has not been carried out on sharpness enhanced portion, strong sharpness enhancing processing results in increase in the granularity to cancel the granularity suppressing processing which has been carried out on the original image signal to obtain the granularity-suppressed image signal. Accordingly, in the case where the operation processing represented by the aforesaid formula (3) is employed, it is preferred that the sharpness enhancing processing disclosed in Japanese Unexamined Patent Publication No. 8(1997)-22460 where the granularity can be suppressed to a certain extent be employed. Further when the sharpness at edge portions is deteriorated by the granularity suppressing processing, this result affects the result of the sharpness enhancing processing, and accordingly, the granularity suppressing processing using morphology operation which can suppress the granularity without deteriorating the sharpness at edge portions is preferred.
Thus it should be appreciated that by employing granularity suppressing processing, such as the granularity suppressing processing using morphology operation which can reserve the sharpness at edge portion, as the processing for suppressing the granularity, and employing the sharpness enhancing processing such as disclosed in Japanese Unexamined Patent Publication No. 8(1997)-22460 where the granularity can be suppressed to a certain extent as the processing for enhancing the sharpness, a more preferable intermediate processed image signal can be obtained.